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he 2002-2003 meetings of the Looking at Student Work project 

(LASW) represent a new phase of the project featuring new organizing principles and 

practices. Previous year’s sessions were observed and documented by EDC/CCT research 

staff and by Center for Arts Education (CAE) staff. Because the 2002-2003 version of the 
project introduced new complexities, the Center administration decided that an outside view would 
be helpful. The reflections presented here were not developed as part of a formal evaluation 
effort, but are those of a “friendly critic,” whose perceptions are informed by long experience with 
the CAE programs and the LASW project. The intent is to provide helpful comments, especially on 
the changes and additions to the original design. As someone has pointed out, simply asking a 
question is often seen as a negative evaluation, and the reader should be cautioned not to make 
that assumption here. It is the obligation of the friendly critic to ask questions as a way of pro- 
voking deliberation and thoughtful future work. Asking questions in this format should not be 
taken as a way of making negative comments, as it is sometimes described. It should be said up 
front that this reviewer has the highest regard for the work done in LASW in 2002-2003. The work 
marks significant gains in the process and in the content from its beginnings, and the contribu- 
tions noted by the participants to their own thinking and classroom work are genuine and impor- 
tant for the field to note. 


The changes implemented in 2002-2003 were made within the framework of goals and purposes 
already established by the project at its inception in 2000. Over the years, “twenty-four sessions, 
each consisting of eight, three-hour workshops, on “Looking at Student Work” were conducted by 
The Center for Arts Education for fifty-four teaching artists, forty-three teachers, and for evalua- 
tors, and Center staff. A total of forty-one partnership projects participated.” 


The form of the sessions in 2002-2003 was consistent with prior years in that they remained an 
informal series of gatherings to review and discuss examples of student work in and through the 
arts. There were ten workshops offered in the fall and spring. In the fall series from November 
2002 through January 2003, thirteen schools registered twenty-five participants including four- 
teen teachers, one school administrator and ten teaching artists. In the Spring series from 
February 2003 to May 2003, eight schools registered twenty-three participants including thirteen 
teachers and ten teaching artists. 


Some invited guest observers attended the sessions. The participants gathered regularly for three 
hours in the afternoon to observe, describe, and discuss examples of student work in an art form 

using one or more protocols. Partnerships were required to send pairs of participants to the ses- 

sions. There were a few sessions when one member of the pair was absent, but for the most part, 
attendance was high. 


Though the 2001-2002 workshops had allowed consideration of more than one art form, the major 
changes in 2002-2003 were to have participants attend separate sessions on Visual Arts, 
Theater/Drama, Dance, and Music instead of having the entire year focused on one or two art 
forms, and the inclusion of a wider variety of protocols for use in observation. As a consequence, 
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specialists in the different art forms were invited to lead the sessions, and the sessions were held 
in facilities related to the art form when possible. CAE did not expect the experts to be fully 
versed in the use of the protocols; the CAE staff was to provide this expertise. 


Although we know that the arts share some common elements and features, each has its own spe- 
cific forms and its own characteristic elements. The unusual combination of uniqueness and sepa- 
rateness causes multi-arts arrangements to be far more complex than those focused on a single 
form or discipline. When we ask participants to “look at student art work,” we must be clear 
about just what kind of “looking” we are asking them to do. The use of the term looking is very 
colloquial in this program and applies widely across very different art forms and differently even 
across the same art form when different performances or presentations are compared. This usage of 
the term is not unusual or incorrect, but some care needs to be taken when more precision is 
needed. Participants in this project would benefit from some practice developing operational defi- 
nitions that apply to the specific cases being observed. They need to define more precisely just 
what they are looking at or looking for in each case, so they can explain to fellow teachers, fellow 
artists, or school administrators. Just how do they “look” at the auditory aspects of a music per- 
formance, for example? How do they look at the use of “space” in a dance and a visual arts pres- 
entation? Teachers may see emotion in a work of visual art differently from the way they see emo- 
tion in theater performances by actors. What exactly were the LASW participants “seeing” when 
they made comparisons of a school dance performance and that of a professional dance company. 
Observers have to come to grips with the added pressures of time and available space when “look- 
ing” at student work in dance performances and when looking at the more static forms of repre- 
sentation in students’ visual arts. Learning how to look both deeply and quickly as the dance per- 
formance passed before their eyes proved to be challenging and enlightening for the participants 
in the LASW workshop. LASW could not provide all the “time and practice” that the dance expert 
advised are needed for anyone to notice what is going on in front of them, but the sessions 
helped participants to become more aware of the needs and the processes that are required for 
observing live performances. 


The complexities of the project's format made it difficult to maintain continuity between 2002- 
2003 LASW sessions and added to the difficulty that the CAE staff faced in making sure that there 
was common understanding and agreement among the presenting experts about what looking in 
their art form was about and about how it should be done. Continuity in the project has been an 
issue from the beginning. The fact that participants and presenters change from year to year 
means that there is little carry over in either experience or gained knowledge. The same questions 
are asked by each year’s participants, as is the case with any beginning level class, but there is lit- 
tle sense of accumulated wisdom or shared expertise among the participants. The project needs a 
more advanced level of participation for those who go through the process. Perhaps the single art 
form version of the project should be the first year, beginning level of experience, and the multi- 
arts version should be offered only to those who have gone through the first level and want to 
move to more complexity in a second year. Thinking about a sequence of experiences over multiple 
years could be profitable. 
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It is interesting that participants in the first year’s class commented on how vague the question. 
“What did you see?” was in the visual arts session, and participants in the 2002-2003 class asked 
the same question. The EDC/CCT researcher that year suggested that another question, “What sort 
of lens are we using to look at student work” might be a better question, but that suggestion was 
not followed up. The same suggestion, broadened to the entire project, still applies. Adding a 
question such as, “What sort of lens are we using?” provides a frame of reference that implies 
operation, function, or activitgy and requires answers that include ways and means, not just 
details. It is the sort of question that addresses the concerns of practitioners who want tools to 
use or activities that can translate into their classroom situations. Adding the question does not 
eliminate the first question, nor does it suggest that the first question is inappropriate, but ask- 
ing the question does imply that answers to it will be sought and theat the project will move to 
more complex levels of thought and operation. What lenses can the project add to the arsenal of 
tools that participants have to take with them? Answers to that question could provide stepping- 
stones to higher levels of deliberation and observation in the project. 


The first year of LASW presented and used Steve Seidel’s Collaborative Assessment Conference 
Protocol to guide the process. In 2002-2003, additional protocols such as those presented by David 
Allen and Tina Blythe in their Looking at Student Work publication were made available to the par- 
ticipants who could choose the protocol(s) they felt most appropriate in their process. The fact 
that theater and dance performances had to be presented in mediated forms added a distance 
between the participants and the observed art that was not present in the visual arts sessions of 
previous years. Music, while it was presented with some direct hands-on experience with musical 
instruments, also required audio or video versions of student work. 


The LASW goal expressed in 2000 was: 


To foster a culture of conversation grounded in student work that contributes to improved stu- 
dent performance and sheds light on curriculum and instruction. To establish a comfort level 
among classroom teachers and teaching artists that might stimulate similar conversations 
within and across schools. 


The goal statement holds true in the current version of the project though the extent to which 
the project “contributes to improved student performance” remains undetermined. This goal state- 
ment includes evaluation terminology in its use of the term “improved, but the project itself does 
not profess to be an evaluation effort nor does it promote an evaluation design process. While the 
goal of contributing to improved student performance is legitimate, without an assessment or 
measurement component attached to the process, there is no way of determining whether this 
part of the goal is met. This issue is a quandary for LASW. It has appeared in the project every 
year, and its resolution is not yet in sight. As teachers, especially, move quickly to judgment and 
evaluative comments, the leaders of the session strive to hold back the tide, asking the partici- 
pants to delay judgment, to wait until they have described more completely, to reflect on what 
they see before they judge it. Every year, 2002-2003 included, sees participants wanting to assess 
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and judge student work, not just observe and describe. The five principles of Seidel’s protocol are: 


e Start by looking at the work knowing little or nothing about the context in which the work 
was produced. 


e Start with description. Withhold judgments as much as possible. 
e It’s fine to come away from the conference with more questions than answers. 
e Keep coming back to the work. 


e Students, when they choose to, invest their work with purpose intent, and an intended audi- 
ence. 


The first principle seems to be a cause of ensuing difficulties for teachers. The development of the 
various protocols for looking at student work over the years has demonstrated the value of doing 
such work, because the teachers benefit from clearer perceptions of students and their work, and 
the students gain from having better informed and experienced teachers and more focused learn- 
ing experiences. At the same time, teachers are taught to know their students and to build their 
instruction upon that knowledge—“to start where the student are.” For them to ignore their con- 
textual knowledge of students and their personal variables violates that part of their practice and 
training and, when they are looking at their own students’ work as part of the LASW process, 
probably causes them to have to pretend not to know the students’ contexts. They can overcome 
this last situation by collaborating with others and by trading student work with other teachers, 
so they truly do not know the students or their context, but school structures and schedules do 
not not permit such exchanges. How can one not have prior knowledge of the students? How can 
one work beyond that knowledge? These questions troubled teachers and add some angst to the 
early LASW sessions. The leaders of the sessions were able to assuage most of the participants’ 
concerns by explaining that the delay in using prior contextual knowledge was not a permanent 
requirement for the process to work. Prior knowledge was allowed in to the process, but only after 
a time of suspended judgment to allow the work to stand on its own. 


The process allows participants to see the value of delayed judgment in the form of new, unexpect- 
ed, or significant insights into the work of students through the protocol process, but LASW also 
needs to consider ways to reassure participants that the practice can be done in their classrooms 
and schools as it is done in LASW workshops and that their contextual knowledge of students 
remains an important part of the process. An important aspect of protocol work is that of collabo- 
ration between teachers. At the end of the session, participants were asked to go back to their 
schools and conduct a brief demonstration LASW session for their fellow teachers. Such sessions 
invite new collaborators in taking the process into the heart of education, the local classrooms. 
Since LASW is expecting participants to make use of these tools in their own schools, it is obligat- 
ed to help them understand how the boundaries that hinder collaboration in schools—using unfa- 
miliar methods or procedures, physical space, grade level assignment, supervisory roles, curriculum 
demands—can be spanned. 
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The issue of collaborative work is another that leads participants into potential conflict with the culture of 
their schools and even of the nation. As a culture of individuals, valuing the individual point of view, the 
individual perspective, the individual set of skills above the communal is paramount. Working in groups 
calls for a willingness to share, cooperate, and even to place a lower value on one’s own individual perspec- 
tive. Seidel’s protocol is called the “Collaborative Assessment Conference Protocol.”(emphasis added.) 
Collaboration implies movement toward consensus, congruence, or convergence of ideas and values. 
Convergence is the easiest of these concepts to accommodate and is the one that LASW should probably 
work toward. Ideas can converge without sacrificing their individual integrity, as consensus sometimes 
requires. For LASW to incorporate this concept in its program, however, staff must reflect on it and the 
practical activities that would foster its development. The development of a common vocabulary—shared 
definitions of terms—often requested by the participants is a step in that direction. 


Participant Responses to the Sessions 


After each session, participants were asked by CAE staff to complete a response form about their 
impressions of the workshop. Participation in the response survey ranged from 60% to 82%, not 
particularly high numbers given the captive nature of the audience, but the late afternoon time 
could explain why others did not stay to respond. The forms in which the questions were asked, 
the changing nature of the responses sought, and the possible conflict between the appearance of 
an evaluation tool being used in a non-evaluative workshop make the use of the data collected 
questionable for any evaluation purposes, but they do cast light on the overall operation of the 
LASW project and are useful for descriptive purposes. The responses reflect the tone of the ses- 
sions and the comments made to or overheard by the observer during the sessions. The presenta- 
tion of the responses as numbers however gives them an authority that they do not have, and 
makes the responses seem more distant and objective than they were. Participants were enthusi- 
astic and actively engaged in the conversations, and their regular attendance indicates a commit- 
ment to the project that was high. The fact that they received a stipend for participation does 
influence their attendance, however. It would be an interesting exercise to work with the session 
leaders to embed the questions into their presentations and gather the data from documentation 
tapes or notes rather than from separate survey instruments. Such a practice would also be a 
demonstration of the kind of assessment that the project asks the participants to develop and use 
in their classes. The number of questions could be reduced in this way with the vague questions 
such as “Did you learn new or useful information?” eliminated. New questions that have concrete 
answers or practical implications such as “Which of the rubrics that eh workshop members devel- 
oped for evaluating student performance in dance are adaptable for the evaluation of music per- 
formance?” “Why?” or “What topics or themes were discussed in students’ integrated journals that 
illustrated their understanding of “dynamics” in music?”. 


Over eighty percent of the fall series participants regularly reported that they gained new and use- 
ful information in the sessions. Details about what was useful were not calculated as percentages. 
These details included: the reaffirmation of their own practice, learning new means of evaluation, 
the process of collaborating with other teachers, using reflection in their practice, adapting the 
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protocols for students, learning new vocabulary and discussing terms related to different art forms, 
the use of new rubric (objectivity, understanding, application) as a means of evaluating dance, 
learning new protocols, using integrated journal writing/cultural research/art experience, learning 
how to use protocols as way to keep classes moving smoothly. 


When asked “how might this way of looking at student work inform your teaching practice” 
responses included: Making better use of rubrics and various measurement tools, and adding the 
language of the several arts into the evaluation and assessment of one art form. It was not clear 
from the response just what this would look like in practice or why it would be done for evalua- 
tion. 


Participants described the protocols as being useful as: Framework/guidelines, providing a starting 
point for discussion of student work, and for creating a comfortable environment. 


Participants reported that the greatest strengths of the Looking at Student Work series included: 
providing the opportunity for them to learn how to do a protocol, the experience of working with 
a variety of disciplines helped in experiencing the use of the protocols in different ways, working 
with a variety of participants, exposure to practice with a variety of assessment tools and proto- 
cols. 


Responses from the spring series were not as positive or as regular as those from the fall series. 
Responses were generally in the less than eighty percent range with the third session getting only 
a fifty percent positive response about whether they gained new and useful information or about 
whether they felt that the protocol was useful for looking at performing arts—dance. The fourth 
session about theatre—playwriting—however got higher numbers for a performing art. One hun- 
dred percent found the information new and/or useful, and eighty-three percent found the proto- 
col useful for looking at performing arts—theatre. 


Among the new and useful elements for the spring participants were: Learning how not to make 
judgments and the value of spending time looking at the work of art, discussing vocabulary terms 
in different art forms, developing new understanding of rubrics (objectivity, understanding, appli- 
cation) and other means of evaluating dance, learning how to use playwriting in schools, learning 
new writing prompts/how to help students write, and learning what the components of a play are. 
Vocabulary/terminology continued to be an important feature as one participant explained, “they 
knew more about music than they thought.” Others learned better use of rubrics and other meas- 
urement tools. They learned to be clearer about goals and criteria and to use comparisons between 
professional dance and children’s dance. The playwriting session was seen as useful for all types of 
writing. 


Ninety-two percent of participants indicated that they liked covering five art disciplines and they 
found equal satisfaction in each discipline’s session. 


Workshop participants saw the greatest strengths of the LASW series as learning how to use a pro- 
tocol; covering a variety of disciplines; using the protocols in different ways; working with a vari- 
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ety of participants from different role groups; and increasing their awareness of different ways of 
evaluating and observing student work. The last item is interesting in that, with the exception of 
the dance workshop, the sessions did not focus on evaluation and the leaders were not experts in 


assessment or evaluation. 


Implications for Action 


There are several courses of action suggested in these reflections. The CAE staff should consider 


convening a planning group to review these suggestions for ways to strengthen an already pol- 
ished program. 


1. 


As the Looking at Student Work project has become more complex, it has given up some inter- 
nal consistency. Working with more than one are form, for example, requires that the protocols 
take into account the different ways that the art forms are presented and set up observation 
activities to take these differences into account. The differences make it difficult to have con- 
sistent results across art forms and may require that separate methods be developed for the dif- 
ferent forms.They may even require different or new or serving protocols. Auditory protocols for 
music, or dynamic space-based protocols for dance may be required instead of the visual obser- 
vation protocols developed for the visual arts. Points of similarity should be maintained and 
emphasized, if only to help convey consistency. Whenever possible, the principles applied 
should be applied to each art, but when the arts have clearly unique features or characteristics, 
then the principles that apply to the art forms must be identified. and used. 


. The practice of having groups attend the sessions as teams should be maintained, but there is 


benefit to be gained from having team members participate with individual from different 
schools and reporting back to their teams and their schools about the variety of approaches 
that are taken and discussed. 


. Workshop session leaders need to be more uniformly familiar with the protocols and use them 


in their sessions. CAE may need to convene some additional common planning times or at least 
have the session leaders meet more with CAE staff to be sure that they are speaking the same 
language about protocols and the approach to looking at student work. 


. The sessions should be focused on looking at student work. When comparisons are made with 


adult artists’ work, the criteria become confusing and the doors are opened to sentimental judg- 
ments about children outweighing quality judgments about art. As long as the comments 
remain descriptive, the issue of judgment is contained, but when adult art/child art compar- 
isons are made, judgment creeps in. 


. Ina related matter, the project needs to work toward joint, clear definitions of the levels of 


work to be considered and the standards, if any to be used. 


. Organizationally, the project leaders need to consider having beginning and more advanced lev- 


els of workshops and extending the timeframe for these sessions to two years. There are differ- 
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ences between having one or two art forms and many art forms as the content for the sessions, 
and there are many more possibilities for considering issues of making judgments about student 
work and about extending this practice toward assessment and evaluation when there is more 
time available. The project need to have a connection to evaluation work and needs to let that 
be explored by the participants throughout the process. 


7. Working with the mediated examples of student work such as videotapes of dance or drama per- 
formances or audiotapes of singing or instrumental music id different from directly observing 
paintings or sculptures. The differences in mediated observations need to be made clear and the 
procedures for such observations worked out with the session leaders. 


8. More explicit work is needed on how to move teachers out of the school culture of making 
judgments about student work based on prior performance or prior knowledge of students’ char- 
acteristics, if they are to make adequate use of the protocols presented in these workshops. 


9. If possible, more sessions need to be held in real schools and classrooms, as they were in the 
beginning sessions. This is only one way of dealing with the need to help the participants, see 
that this practice can take place in their own classes, and that it relates to the real work that 
they do. 


10.The practice of asking participants to demonstrate the LASW process back in their home schools 
at the end of the workshop series could become a crucial dissemination element, especially if 
there is some modification to the assignment. There is no time to reflect or get feedback on 
how these demonstrations work, if they take place after the workshop is over. Moving the 
assignment up in time and asking for some documentation of the demonstration would allow 
the participants and CAE to learn from the experience and could provide information that could 
be used to strengthen the workshop sessions. 


11.It would help for the project to solicit and present to each new group of participants some 
endorsement of the collaborative practice used in these sessions from school administrators or 
from past participants. 


